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Abstract 

Bibliometric benchmarking can be an aid to researchers pondering whether to apply for 
competitive grants. In this paper, the highly prestigious grants offered by the European 
Research Council to young scientists of any nationality were scrutinized. The analysis of the 
2014-2015 data indicates that over 75% of life science grantees in the starting category (2-7 
years after completing a Ph.D. degree program) had at least 14 papers and an H-index of 10 
(28 and 16, respectively ,in the case of the consolidator category—i.e., 7-12 years after 
obtaining the Ph.D.). Yet other signs of excellence, expert advice, and the limitations of metric 
approaches need to be considered. 


Introduction 

Think of a scientist pondering whether 
to submit an application to the European 
Research Council (ERC). The carrot is a 
tasty one: joining the prestigious club of 
those—over 5,500 members of 66 
nationalities, including over 180 U.S. 
nationals—who have received, since 2007, 
one of the 5-year 1.5-2.5 million euro grant 
(European Research Council, 2015). The 
scientist may have an excellent project idea 
(ground-breaking and high-gain high-risk. 


in ERC language), which will be put to the 
test during the evaluation process. 
However, a second criterion will have to be 
met: the CV and track record also must be 
excellent. So what sort of measure of 
excellence would that be? After all, even for 
the best scientist at a given department or 
institution, a track record that is not 
perceived as outstanding at the European 
level will have no chance, meaning months 
of preparation spent in vain. The opposite 
may also occur, with scientists who could 
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potentially succeed not submitting an 
application due to excessive modesty. 

Benchmarking merits against previous 
grantees may be the next thought—in most 
cases these are young scientists either 
within the starting (2-7 years after Ph.D.) or 
consolidator (7-12 years after Ph.D.) 
categories (European Research Council, 
2016). And here is where, at least within 
some disciplines (including the life 
sciences), bibliometric analyses could prove 
handy. So with a focus on the ERC Life 
Science (LS) domain and its 9 panels, key 
publication metrics for all 488 Starting and 
Consolidator grantees in the 2014 and 2015 


rounds were retrieved from Scopus. Figure 
1 displays quartile values, which may offer 
a useful reference to the pondering young 
scientist while controlling for outliers. The 
highlighted first quartile values, for 
example for the LSI panel (Molecular and 
Structural Biology and Biochemistry), reveal 
that 75% of grantees in the starting category 
had at least 13 papers (counted as Scopus- 
type articles and reviews), 523 citations (not 
shown) and H-index of 10. The equivalent 
values for consolidators in the same panel 
are 24 papers, 949 citations, and an H-index 
of 16. 


(A) No. of papers, ERC StG grantees 2014-2015 (B) No. of papers, ERC CoG grantees 2014-2015 



• Median — ■ — Q1 Max Min A Q3 • Median — ■ — Q1 Max Min A Q3 


(C) H-indices, ERC StG grantees 2014-2015 (D) H-indices, ERC CoG grantees 2014-2015 



LS all LSI LS2 LS3 LS4 LS5 LS6 LS7 LS8 LS9 LS all LSI LS2 LS3 LS4 LS5 LS6 LS7 LS8 LS9 

• Median — ■ — Ql Max Min A Q3 • Median — ■ — Q1 Max Min A Q3 

Figure 1. Bibliometrics for the 2014-2015 ERC Starting and Consolidator Grantees in the life 
Sciences. (A) and (B) display number of papers and (C) and (D) H-indices. Value ranges are 
indicated by vertical lines. Boxes delimit the first (Ql) and third (Q3) quartiles, with median 
values also marked. Ql values are shown and connected by a line. 
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The graphs also capture differences 
between panels, likely reflecting variation in 
publication and citation practices by field. 
Values were somewhat lower in panels such 
as LS3 (Cellular and Developmental 
Biology) and LS5 (Neurosciences and 
Neural Disorders). The LS7 panel 
(Diagnostic Tools, Therapies and Public 
Health) showed some of the highest values, 
possibly consistent with the fact that this life 
science panel received the most 
applications. The highest interquartile range 
values were found in some of the 
consolidator panels (e.g., LS7 and LS4— 
Physiology, Pathophysiology and 
Endocrinology), indicating a higher 
variability in grantees' metrics. 

ERC Myths and Facts 

While little has been shared in the 
literature, there is a great deal of expertise 
on the ERC among European research 
institutions. In the words of the late 
Professor Ilkka Hanski (ERC Advanced 
grantee and former panel chair), an H-index 
approaching 10 could be considered a good 
indication when applying for ERC Starting, 
which seems in line with the values shown 
above. With respect to the often claimed 
"hidden" additional criteria, such as the 
need for prior international mobility, 
preliminary evidence reassuring feasibility, 
or a paper in Science, Nature or Cell (The 
Guardian, 2014), no evidence was found to 


support such claims. With regard to the 
latter, the data showed tremendous panel 
variations, with, for example, 80% of 
starting grantees in LSI having such a paper 
versus just 18% in LS7. Considering the 
tough international competition for these 
grants, these and any other merits are 
simply likely to help—for example, it has 
been observed that ERC panels tend to 
select applicants who have published high- 
impact articles (Robitaille et al., 2015). 
Without splitting hairs about the values 
shown in the figure, we could derive the 
general conclusion that young scientists 
whose metrics are above first quartile 
values should probably stop pondering and 
start applying. 

All things considered, this look into 
metrics confirms something that comes as 
no surprise: the ERC is highly competitive 
and only for the best, with the 12-15% 
success rates for 2015 as a reminder. Yet, 
metrics could be of use in planning and 
rationalizing efforts when targeting 
research funding. The fairness of the ERC 
process [aside from political considerations 
(The Guardian, 2014) or exasperation about 
ever-changing deadlines] is widely 
acknowledged, especially with regard to 
recognizing excellence. Not surprisingly, 
rankings of excellence nowadays include 
counts of ERC grants, and an increasing 
number of organizations are re-modelling 
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their schemes to imitate the ERC, with even 
some national programs sponsoring the best 
non-funded ERC applications ( Nature Cell 
Biology, 2010). As institutional recruitment 
practices follow suit, the issue at hand will 
continue to attract attention. 

A recent report commissioned by the 
ERC indicated that scores attributed by the 
evaluation committees match well with 
applicant performance as measured by 
bibliometric indicators (Robitaille et al., 

2015, p. 69). Still, obvious caution should be 
exercised in the interpretation of our values 
given the limitations of the metrics 
approach [see Science (2016) for a discussion 
on metrics for young scientists]. In the case 
presented here, only grantees' profiles were 
considered, with the excellence of the 
project idea criterion not accounted for. 
While effort was put into cross-checking the 
accuracy of the data (e.g., checking 
researchers' ORCID and their own websites 
when necessary), the publication numbers 
in Scopus may not exactly match the 
scientists' record at the time of the 


evaluation. Furthermore, the metrics 
analyses only capture a part of the picture 
(e.g., overlooking merits such as awards or 
the scientist's contributions to multi- 
authored papers). Whereas substantial 
differences in the material and methods 
used impeded a comparison of results with 
two related studies [those of Pecha on a 
2012 Starting grantees' cohort (Pecha, 2014), 
and of the MERCI project with 2007 and 
2009 Starting applicants (MERCI project, 
n.d.)], the described analyses can be easily 
replicated with future ERC cohorts, and 
may be likewise relevant for other 
competitive research funding schemes. 
When interpreting values, the golden rules 
of metrics also need to be remembered: use 
more than one metric to give insights into 
an issue, and support conclusions with 
expert analysis (Colledge & Verlinde, 2014). 
Research advisors, ERC National Contact 
Points, and particularly ERC-experienced 
scientists will be great supports for the 
pondering scientist who, if still doubting, 
should just be encouraged to apply. 
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